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FINAL  REPORT  ON'  AN  EXPERIMENTAL  INVESTIGATION  OF  THE  EFFECTS  OF 
TECHNICAL  REPORT  ABSTRACTS  AND  OTHER  EXPERIMENTAL  VARIABLES 
ON  THE  BEHAVIOR  OF  INFORMATION  USERS 


The  purpose  of  this  paper  is  to  report  on  a  field  experiment  which  is  part 
of  a  long  term  series  of  investigations  of  the  information  related  behavior 
or  engineers  and  scientists.  Specifically,  the  report  focuses  on  phases  of 
the  research  which  have  been  supported  by  the  Office  of  Naval  Research 
(Information  Systems  Branch,  Mathematical  Sciei.-es  Division). 


Scope  of  the  Study 


Since  1959,  Northwestern  University  has  had,  under  the  direction  of  Professor 
Albert  H.  Rubenstein,  a  Program  of  Research  on  the  Management  of  Research  and 
Development.  The  program  consists  of  approximately  a  dozen  individual  pro¬ 
jects  or  subprograms,  focused  on  specific  aspects  of  the  R&D  process.  Among 
the  subprograms  (each  consisting  of  a  continuing  series  of  staff  studies  and 
graduate  theses)  are  studies  of  the  effects  of  decentralization  on  R&D, 
organization  of  R&D  in  newly  developing  countries,  the  generation  and  flow 
of  ideas  in  R&D,  project  selection  in  R&D,  and  the  information  seeking 
behavior  of  researchers. 


The  objective  of  that  part  of  the  latter  study  which  is  reported  here  was 
"to  investigate,  by  means  of  field  experiments  in  operating  R&T  laboratories, 
behavioral  effects  of  the  form  and  use  of  abstracts  of  technical  reports 
c  .d  other  variations  in  the  information  environment  of  researchers."  In 
pursuit  of  this  objective,  two  phases  were  outlined,  and  these,  together 
with  a  brief  description  of  what  was  done,  will  follow. 


Phase  One 

In  late  1966,  prior  to  the  beginning  Phase  One,  preliminiry  discussions 
between  A.  H.  Rubenstein,  the  Principal  Investigator,  and  R.  H.  Wilcox,  of 
the  Office  of  Naval  Research,  took  place  concerning  the  general  feasibility 
of  exploring,  by  field  experimental  means,  the  behavioral  effects  of  the  form 
and  use  of  abe tracts  of  technical  reports.  In  the  early  spring  of  1967  a 
preliminary  de_ii,  '  was  reached  to  explore  the  behavioral  effects  of 
abstracts  in  one  or  more  military  laboratories,  and,  based  on  this,  a  major 
literature  search  was  started,  a  set  of  preliminary  design  notes  was  pre¬ 
pared  describing  the  proposed  field  setting,  outlining  the  research  design 
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(which  was  the  design  eventually  used),  listing  a  number  of  parameters,  and 
stating  the  objective  of  an  initial  proposition  (which  became  the  first 
proposition  discussed  below).  During  the  summer  of  1967,  prospective  sites 
were  visited  and  further  preliminary  designs  were  explored,  leading  up  to 
the  beginning  of  Phase  One  in  the  fall  of  1967. 

Phase  One  covered  the  design  and  pilot  testing  of  the  field  experiment,  in 
five  steps,  as  follows: 

a)  An  analysis  of  the  area  of  research  interest,  based  in  part  on 
a  collection  of  approximately  700  reports  of  research  and  the 
observation  of  knowledgeable  practitioners,  was  accomplished; 
and  this  resulted  in  an  appraisal  of  the  state-of-the-art, 
including  available  methodology,  the  problems  involved,  and 
the  potential  implications  of  several  alternative  research 
strategies.  The  most  significant  general  findings,  neither  of 
which  was  wholly  unexpected,  were  that  little  is  known,  in  a 
-scientific  sense,  about  the  behavioral  effects  of  the  form  of 
a  document  (such  as  the  presence  or  absence  of  an  abstract), 

and  that  this  may  be,  in  part,  due  to  the  second  finding  that  the 
methodology  to  examine  these  effects  under  field  conditions  has 
not  been  substantially  developed.  As  a  corollary,  it  appeared 
that  the  "screening"  process  which  occurs  when  a  document  is 
first  physically  available  to  the  individual  scientist  or  engineer 
is  central,  from  a  theoretical  point  of  view,  to  not  only  a  wide 
variety  of  behavioral  research  areas  but  also  research  into  the 
design  of  information  retrieval  systems.  This  centrality 
suggested  3  significant  potential  value  for  the  research. 

b)  Based  on  this  analysis,  a  set  of  researchable  questions  was 
developed,  and  a  selection  was  made  in  consideration  of  the 
advantages  and  limitations  presented  by  use  of  field  experi¬ 
mental  means,  and  the  potential  implications  of  the  results  of 
the  research.  This  set  included  the  following  questions: 

-  Doec  an  abstract  perform  any  significant  function  when 
the  document  itself  is  available? 


-  What  effects  do  variations  in  form  (length,  location  in  j 

the  document,  identifying  features,  etc.)  of  an  abstract 

have  upon  the  disposition  decision? 

-  What  effects  do  variations  in  content  (descriptive  versus 
informative,  "quality,"  etc.)  of  an  abstract  have? 

-  What  effects  do  variations  in  the  background  (age,  sex, 
technical  specialty,  experience,  level  of  competence ,  etc.) 
of  an  individual  have  upon  the  relation  between  an  abstract 
and  his  disposition  decision? 

f 

-  What  effects  do  variations  in  his  immediate  environment  ! 

i 

(organizational  policies,  procedures;  task;  various 
available  alternative  facilities,  such  as  personal  files, 
libraries,  abstracting  and  indexing  services,  colleagues; 
characteristics  of  the  queue;  personal  health;  time  of 
day;  etc.)  have? 

-  What  effects  do  variations  in  his  immediate  purpose  (current 
awareness  versus  search,  information-seeking  versus  "clearing 
his  desk"  or  "make  work")  have? 

From  these  questions,  two  additional  question,  which  were  based  on  the  first 
question,  were  developed.  These  two  questions,  which  provided  the  basis  for 
the  research,  were  as  follows: 

-  What  relationship  is  there  between  the  presence  or  absence 
of  an  abstract  and  the  amount  of  time  taken  to  make  a 
disposition  decision? 

-  What  relationship  is  there  between  the  presence  or  absence 
of  an  abstract  and  the  "correctness"  of  the  disposition 

decision  (the  "correctness"  of  the  determination  of  , 

relevance)?  j 

i 

I 

c)  From  these  two  questions,  two  central  propositions,  together  with 
a  number  of  corollary  propositions,  were  developed,  and  they  were 

i 

as  follows: 

-  For  all  documents  which  arc  not  disposed  of  on  first  ; 

I 

glance,  the  disposition  time  for  those  documents  accomoanied 
by  an  abstract  is  significantly  shorter  than  for  those  which 

are  not.  j 

i 


-  For  all  documents  which  are  not  strongly  relevant  (or 
strongly  irrelevant),  the  ability  to  judge  relevance 
is  significantly  better  for  those  documents  accompanied 
by  an  abstract  than  for  those  which  are  not. 

d)  For  each  proposition,  definitions  were  developed,  indicators  of  the 
variables  of  interest  were  identified,  parameters  were  established, 
in  terms  of  measurement  or  control,  and  the  design  of  the  experi¬ 
ment  was  outlined. 

e)  Research  instruments,  with  several  variations,  were  designed, 
pretested  several  times,  and  then  tested  in  a  pilot  site. 

Phase  Two 

The  conduct  of  the  field  portion  of  the  research  was  accomplished  in  three 
Navy  laboratories  in  the  Washington  area  over  the  period  from  the  middle  of 
July  to  the  end  of  September,  1968,  through  the  cooperation  of  ONR  and  with 
the  active  assistance  of  a  key  staff  member  in  each  laboratory.  Negotiations 
to  obtain  entry  took  place  over  a  period  of  several  months  and  were  success¬ 
ful  particularly  due  to  the  extended  efforts  by  ONR.  Through  agreement  with 
senior  management  of  the  laboratories,  the  following  arrangements  were  made: 

a)  Each  laboratory  would  provide  a  staff  member  who  would  (select  and) 
make  arrangements  for  participants,  and  for  use  of  library  facilities 
by  the  Northwestern  research  team. 

b)  Each  laboratory  would  provide  participants  (25  in  one  laboratory, 
and  30  in  each  of  the  other  two)  representing  a  rough  sample  of 
the  scientific  and  engineering  population  of  the  laboratory  in 
terms  of  organizational  divisions,  the  science-to-engineering 
spectrum,  disciplines  and  seniority  (experience  and  position  held). 

c)  The  university  researcher  designed  the  field  experiment  to  limit 
the  involvement  of  participants  as  follows: 

( 1 )  An  initial  briefing,  not  to  exceed  thirty  uifout'**' 


(2)  A  four  week  research  period,  not  to  exceed  a  total  of 
30  minutes  by  each  individual,  to  complete  a  maximum  of 
100  short  forms. 

(3)  A  post  interview  not  tc  exceed  one  hour. 

(A)  An  overall  time  limit  of  three  hours. 

The  initial  briefings  took  place  in  late  July  and  early  August,  1968.  Each 
was  within  the  30  minute  period.  Each  participant  was  provided  with  50 
copies  of  a  one-page  form  ar.d  asked  to  fill  it  cut  for  each  document  (defined 
primarily  as  a  technical  report  or  journal  article)  which  came  across  his 
desk  during  the  research  period.  The  key  questions  related  to  the  time  he 
took  to  decide  what  to  oo  with  the  report  or  journal  article  end  the  relevance 
judgment  he  made.  During  the  research  period  of  four  weeks,  the  university 
researchers  made  weekly  visits  to  each  laboratory  to  pick  up  data,  to  begin 
the  library  search  for  variables  data  on  reports  and  journals,  and  to  answer 
questions  raised  by  the  participants. 

At  the  end  of  the  research  period,  each  participant  was  requested  to  complete 
a  questionnaire  on  parametric  and  background  questions  which  would  assist  in 
understanding  the  variables  data.  In  addition,  participants  identified 
according  to  rules  established  by  the  research  design  were  interviewed  in 
person  and  asked  to  repeat  the  relevance  judgment  they  had  made  previously 
with  respect  to  selected  documents. 

All  of  the  data  with  respect  to  the  participants'  decisions  during  the 
research  period  was  key-punched  after  accession  into  the  formal  central 
files  maintained  for  all  research  data  in  the  Program  of  Research  on  the 
Management  of  Research  and  Development.  In  addition,  all  available  docu¬ 
ments  for  which  dependent  variable  data  meeting  the  propositional  require¬ 
ments  was  obtained  were  analysed  for  the  presence  or  absence  of  an  abstract 
(and  for  a  variety  of  parametric  data);  these  data  were  also  key-punched. 

In  all,  71  persons  in  the  three  laboratories  completed  950  individual  forms, 
and  for  these.  391  documents  were  analysed  for  the  presence  or  absence  of 
abstracts  ;  73  persons  completed  Post  Test  Questionnaires,  and  13  persons 


-5- 


were  interviewed.  One  senior  individual  who  had  provided  extensive  valuable 
data  graciously  agreed  to  a  second  interview;  this  was  accomplished  within 
the  agreed  one  hour  time  limit.  Twenty-two  individuals  provided  data 
sufficient  to  test  the  first  proposition  on  time  taken,  and  twelve  indivi¬ 
duals  provided  data  sufficient  to  test  the  second  proposition  on  relevance. 

Analysis  of  each  set  of  individual  data  and  analysis  of  the  aggregate  pro¬ 
vided  no  support  for  either  proposition.  While  it  is  not  considered  war¬ 
ranted  to  assert  that  the  null  hypothesis,  i.e.,  the  presence  or  absence 
of  an  abstract  does  not  make  a  difference,  was  proved,  it  seems  clear  that 
a  number  of  "independent"  opportunities  were  presented  for  the  propositions 
to  be  supported,  and  in  no  instance  did  support  appear.  It  is  concluded  that 
the  interrelation  of  a  variety  of  variables  must  be  accounted  for  in  greater 
detail  before  any  strong  statements  concerning  the  effect  of  abstracts  can 
be  demonstrated.  Whatever  may  be  the  function  and  effects  of  abstracts  under 
laboratory  conditions,  as  described  in  the  literature,  in  the  normal,  day- 
to-day  field  setting,  the  presence  of  an  abstract  does  not  appear  to  be  a 
significant  factor  in  the  determination  of  what  disposition  to  make  of  a 
document . 
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I  1  T 


The  field  study  (or  "natural  experiment")  was  conducted  in  thr^u  military  laboratories 
to  determine  what  relationship  there  is  between  the  presence  or  absen<-_  of  an  abstract 
and  the-  time  required  to  make,  and  "correctness"  of,  the  decision  an  how  to  dispose  of 
a  technical  report  or  -tide  which  arrives  at  the  d°r  ;  of  an  individual 

scientist  or  engineer.  The  measure  of  time-  wac  a  sr'  '-.eyort  by  the  individual:  the 
measure  of  "correctness"  war  the  degree  of  agreement  between  the  self-report  of  "use¬ 
fulness"  made  at  the  tine  of  disposition  and  a  later,  deliberate  evaluation  of  "use¬ 
fulness  by  the  sine  individual.  An  arbitrary  sample  of  S5  persons  was  chosen.  Seventy- 
eight  participated  to  some  degree,  some1  minimum  usable  data  was  received  from  71,  and 
significant  amounts  of  data  f^om  22.  For  a  four  vci  k  period,  each  participant  was 
asked  to  report  oa  the  time  taken  to  dispose  of,  and  the  "usefulness"  of,  each  tech¬ 
nical  document  he  encountered,  at  the  time  oi"  the  encounter.  I  he  researcher  later 
retrieved  the  documents  and  determined  if  an  abstract  wire  present,  and  noted  certain 
other  characteristics.  Those  participants  whose  data  met  certain  conditions  oi  the 
study  were  then  presented  with  selected  documents  which  they  had  nrcviouslv  encount¬ 
ered,  and  were  requested  to  again  note  the  "usefulness."  These  participants  were  I 

observed  during  the  process  of  this  encounter  with  the  documents,  and  were  interviewed. 
All  participants  completed  a  questionnaire  on  parametric  and  background  information. 
With  respect  to  the  relationship  to  time,  the  data  does  not  support  the  proposition 
that  there  is  a  strong  and  clear  relation.  With  respect  to  "correctness"  the  same 
applies.  It  is  concluded  that  the  interrelation  of  a  variety  of  variables  must  be 
accounted  for  in  greater  detail  before  any  strong  statements  concerning  the  effect  of 
abstracts  can  be  demonstrated.  Whatever  ray  be  the  function  and  effect  of  abstracts 


Continuation  of  Abstract 


under  laboratory  conditions,  in  the  normal,  day-to-day  field  setting,  the 
presence  of  an  abstract  does  not  appear  to  be  a  significant  factor  in  the 
determination  of  what  disposition  to  make  of  a  document. 


